CI: unpin pyarrow, fix failing test #51175
Conversation
pandas/core/arrays/arrow/array.py
Outdated
| if isinstance(strings, (pa.Array, pa.ChunkedArray)): | ||
| scalars = strings | ||
| else: | ||
| scalars = pa.array(strings, type=pa.string(), from_pandas=True) |
There was a problem hiding this comment.
IMO (slightly) I think we should still prefer to use to_timedelta first if a non-pyarrow scalars was passed over the cast
There was a problem hiding this comment.
I think using to_timedelta first would be incorrect in some cases. For instance, if passed "1000 seconds" as
ArrowExtensionArray._from_sequence_of_strings(["1000"], dtype=pa.duration("s"))
pd.to_timedelta(["1000"]) is interpreted as 1000 ns (to_timedelta does not allow specifying unit with a string input and assumes nanoseconds).
There was a problem hiding this comment.
in fact, the example provided above errors on main due to assuming ns units:
import pyarrow as pa
ArrowExtensionArray._from_sequence_of_strings(["1000"], dtype=pa.duration("s"))
# ArrowInvalid: Casting from duration[ns] to duration[s] would lose data: 1000
There was a problem hiding this comment.
FWIW there's a corner case with iNaT that pyarrow will handle differently from to_timedelta
There was a problem hiding this comment.
Good point re: NaT. I made some changes which now call to_timedelta first and only attempts the int conversion when the pyarrow unit is non-nano. It respects the null mask from to_timedelta so I think the NaT edge case is handled. I added a test for it.
It all feels a bit messy, but I don't see another option ATM. Open to suggestions.
|
Thanks @lukemanley |
|
I guess we should add 11 to our build matrix? |
|
Good point. Otherwise 11 is only running on the macos and windows builds. Want to open a PR? |
* unpin pyarrow, fix failing test * cleanup * handle NaT/NaN
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.Unpins pyarrow and fixes
test_from_sequence_of_strings_pa_arraywhich was xfailed for pyarrow duration types.pyarrow 11.0 introduces support for casting from duration to string:
pyarrow < 11.0 raises: